{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# [HW2] Practice With Augmented Regression\n",
                "\n",
                "Import necessary Python packages and seed the random number generator.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import numpy as np\n",
                "import matplotlib.pyplot as plt\n",
                "plt.rcParams['figure.figsize'] = 10, 6\n",
                "np.random.seed(174515)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Helper Functions\n",
                "\n",
                "**Fill in these functions using your code from the hyperparameter tuning problem on this homework.**\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def ridge_regress(X, y, lambd):\n",
                "    ### start ridge ###\n",
                "\n",
                "    ### end ridge ###\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def optimize_lambda(X_train, y_train, X_val, y_val, candidate_lambdas=np.logspace(-3, 0, 50)):\n",
                "    ### start Optimize_Lambda ###\n",
                "\n",
                "    ### end Optimize_Lambda ###\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def plot_validation_errors(X_train, y_train, X_val, y_val, candidate_lambdas=np.logspace(-3, 0, 50)):\n",
                "    # This version of the function only plots the validation error, since we don't know the true\n",
                "    # vector w.\n",
                "    w_mses = []\n",
                "    val_errors = []\n",
                "    for l in candidate_lambdas:\n",
                "        ### start Compute_Errors ###\n",
                "\n",
                "        ### end Compute_Errors ###\n",
                "\n",
                "    plt.figure()\n",
                "    plt.title(\"Error vs $\\lambda$\")\n",
                "    plt.yscale('log')\n",
                "    plt.xscale('log')\n",
                "    plt.xlabel(\"$\\lambda$\")\n",
                "    plt.ylabel('Estimation Error')  # we already handled the x-label with ax1\n",
                "    plt.plot(candidate_lambdas, val_errors)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Load the Training Data\n",
                "\n",
                "This dataset uses physiochemical features of white wines to predict an expert's quality score. It has been lightly preprocessed to normalize the features and labels. As you will see later, preprocessing can be extrememly important for model performance. You can access the original data in the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Wine+Quality).\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "X = np.load(\"wine_train_features.npy\")\n",
                "y = np.load(\"wine_train_labels.npy\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Part (a): Explicit Ridge Regression\n",
                "\n",
                "First, we will perform explicit ridge regression and tune $\\lambda$ to get a baseline for comparison.\n",
                "\n",
                "**Generate an 80-20 training/validation split using the loaded data.**\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "print(\"All data:\", X.shape)\n",
                "n, d = X.shape\n",
                "perm = np.random.permutation(n)\n",
                "ntrain = int(n * 0.5)\n",
                "idx_train = perm[:ntrain]\n",
                "idx_val = perm[ntrain:]\n",
                "X_train = X[idx_train, :]\n",
                "y_train = y[idx_train]\n",
                "X_val = X[idx_val, :]\n",
                "y_val = y[idx_val]\n",
                "print(\"Training data:\", X_train.shape)\n",
                "print(\"Validation data:\", X_val.shape)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Use `plot_validation_errors` and `optimize_lambda` to tune $\\lambda$ for use in the rest of this problem.** Try running for several train/validation splits to see how $\\lambda$ varies. Pick a reasonable value.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# TODO: Find a reasonable lambda\n",
                "plot_validation_errors(X_train, y_train, X_val, y_val, candidate_lambdas=np.logspace(-2, 3, 100))\n",
                "lambd = optimize_lambda(X_train, y_train, X_val, y_val, candidate_lambdas=np.logspace(-2, 3, 100))\n",
                "print(\"Tuned lambda: %f\" % lambd)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Use your tuned $\\lambda$ to find $\\hat{w}$ by regressing on all the data. Print the MSE.**\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "### start w_hat_rr ###\n",
                "\n",
                "### end w_hat_rr ###\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Part (b): Regularization with Augmented Features\n",
                "\n",
                "Here you will train a regressor with augmented features. Refer to 6(c) from HW1 for context.\n",
                "\n",
                "First **augment the training data.**\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# TODO: Augment the training data in feature space.\n",
                "# Use the appropriate weight for the augmentation identity.\n",
                "d_raw = X.shape[1]\n",
                "print(\"X shape\", X.shape)\n",
                "X_aug = np.zeros((X.shape[0], X.shape[1] + n))\n",
                "print(\"X augmented shape\", X_aug.shape)\n",
                "### start b1 ###\n",
                "\n",
                "### end b1 ###\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Next, **perform the minimum-norm least-squares optimization and report the MSE.** The plot of $\\hat{w}$ calculated in both ways should be one overlapping line (identical results).\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# TODO: Solve the least-norm linear regression problem. Print the\n",
                "# residual squared error.\n",
                "\n",
                "# Be careful! You are working with an underdetermined system so you\n",
                "# will need to modify the problem you solve from the overdetermined\n",
                "# least-squares case.\n",
                "# This can be solved using SVD, however explicitly calculating the SVD\n",
                "# for large matrices takes a LONG time. You should think about\n",
                "# how to do this regression in a more efficient way.\n",
                "\n",
                "# Hint: Read and understand https://see.stanford.edu/materials/lsoeldsee263/08-min-norm.pdf\n",
                "\n",
                "### start b2 ###\n",
                "\n",
                "### end b2 ###\n",
                "\n",
                "# Plot w_hat calculated both ways. They should be identical.\n",
                "plt.plot(w_rr, label='$\\hat{w}$ ridge regression')\n",
                "plt.plot(eta[:d_raw], label='$\\hat{w}$ feature augmentation')\n",
                "plt.legend();\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Plot $\\eta$ and the subset of $\\eta$ corresponding to the original features.\n",
                "Interpret the residual squared error in the context of the augmented dataset.**\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# TODO: Plot eta and the subset of eta corresponding to the original features.\n",
                "#       Interpret the residual squared error in the context of the augmented dataset.\n",
                "### start b3 ###\n",
                "\n",
                "### end b3 ###\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "_Your interpretation of the squared error here_\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Part (c): Predicting with Augmented Features\n",
                "We have two methods available for prediction. We can extract only the parts of $\\eta$ corresponding to $\\hat{w}$, or we can appropriately augment the test set and use some other subset of $\\eta$ for prediction. Here you will use both methods and see that they give the same results for test prediction.\n",
                "\n",
                "First, **predict using only the part of $\\eta$ corresponding to $w$. Report the train and test MSEs.**\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Load the test dataset\n",
                "# It is good practice to do this after the training has been\n",
                "# completed to make sure that no training happens on the test\n",
                "# set!\n",
                "X_test = np.load(\"wine_test_features.npy\")\n",
                "y_test = np.load(\"wine_test_labels.npy\")\n",
                "\n",
                "n_test = y_test.shape[0]\n",
                "\n",
                "# TODO: Use the w_hat subset of eta and calculate the train and test MSE\n",
                "\n",
                "# First method: extract only w from eta and use it\n",
                "### start c1 ###\n",
                "\n",
                "### end c1 ###\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Next, **appropriately augment the test set, and predict using the appropriate subset of $\\eta$. Report train and test MSEs.**\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Second method: augment test set features\n",
                "# -> What should we use for the augmented features?\n",
                "### start c2 ###\n",
                "\n",
                "### end c2 ###\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Part (d):\n",
                "What if we add small random weights on the diagonal instead of 0s to the test set?\n",
                "\n",
                "**Choose a few distributions and plot the test MSE for augmentation with a range of scalings of the distribution.**\n",
                "One example might be\n",
                "```\n",
                "for u in np.linspace(0, 1, 50):\n",
                "    test_augmented[:, d_raw:] = np.diag(np.random.uniform(0, u, n))\n",
                "    ...\n",
                "```\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# TODO: instead of 0s for test features, use something small and random on the diagonal.\n",
                "test_mses = []\n",
                "scalings = []\n",
                "### start d ###\n",
                "\n",
                "### end d ###\n",
                "plt.plot(scalings, test_mses)\n",
                "plt.title(\"MSE with random augmentations\")\n",
                "plt.xscale('log')\n",
                "plt.xlabel('Augmentation Scaling$')\n",
                "plt.ylabel('MSE');\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Comment on what you observe about the effect of distribution and scaling on the test performance.**\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "_Your comments here_\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": []
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.8.2"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 4
}